{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contract Analysis Using 6B Model on Laptop with LLMWare"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imagine you were given a large contract and asked a really specific question about it: \"What is the notice for termination for convenience?\" It would be an ordeal to locate the answer for this in the contract.\n",
    "\n",
    "But what if we could use AI 🤖 to analyze the contract and answer this for us?\n",
    "\n",
    "What we want here is to perform something known as retrieval-augmented generation (RAG). This is the process by which we give a language model some external sources (such as a contract). The external sources are intended to enhance the model's context, giving it a more comprehensive understanding of a topic. The model should then give us more accurate responses to the questions we ask it on the topic.\n",
    "\n",
    "Now, a general purpose model like Chat-GPT might be able to answer questions about contracts with RAG, but we can do something better. We can use LLMWare's dragon-yi-6b-gguf model, which is RAG-finetuned for fact-based question-answering on complex business and legal documents."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For Google Colab Users"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are using Colab for free, we highly recommend you activate the T4 GPU hardware accelerator. Our models are designed to run with at least 16GB of RAM, activating T4 will grant the notebook 16GB of GDDR6 RAM as apposed to the ~13GB Colab gives automatically.\n",
    "\n",
    "To activate T4:\n",
    "1. click on the \"Runtime\" tab\n",
    "2. click on \"Change runtime type\"\n",
    "3. select T4 GPU under Hardware Accelerator\n",
    "\n",
    "NOTE: there is a weekly usage limit on using T4 for free"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing and importing dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Defaulting to user installation because normal site-packages is not writeable\n",
      "Requirement already satisfied: llmware in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (0.2.14)\n",
      "Requirement already satisfied: boto3==1.24.53 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from llmware) (1.24.53)\n",
      "Requirement already satisfied: huggingface-hub>=0.19.4 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from llmware) (0.19.4)\n",
      "Requirement already satisfied: numpy>=1.23.2 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from llmware) (1.26.0)\n",
      "Requirement already satisfied: openai>=1.0.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from llmware) (1.13.3)\n",
      "Requirement already satisfied: pymongo>=4.7.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from llmware) (4.7.2)\n",
      "Requirement already satisfied: tokenizers>=0.15.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from llmware) (0.15.2)\n",
      "Requirement already satisfied: torch>=1.13.1 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from llmware) (2.1.0+cu121)\n",
      "Requirement already satisfied: transformers>=4.36.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from llmware) (4.38.2)\n",
      "Requirement already satisfied: Wikipedia-API==0.6.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from llmware) (0.6.0)\n",
      "Requirement already satisfied: psycopg-binary==3.1.17 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from llmware) (3.1.17)\n",
      "Requirement already satisfied: psycopg==3.1.17 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from llmware) (3.1.17)\n",
      "Requirement already satisfied: pgvector==0.2.4 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from llmware) (0.2.4)\n",
      "Requirement already satisfied: colorama==0.4.6 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from llmware) (0.4.6)\n",
      "Requirement already satisfied: einops==0.7.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from llmware) (0.7.0)\n",
      "Requirement already satisfied: librosa>=0.10.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from llmware) (0.10.2.post1)\n",
      "Requirement already satisfied: botocore<1.28.0,>=1.27.53 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from boto3==1.24.53->llmware) (1.27.96)\n",
      "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from boto3==1.24.53->llmware) (1.0.1)\n",
      "Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from boto3==1.24.53->llmware) (0.6.2)\n",
      "Requirement already satisfied: typing-extensions>=4.1 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from psycopg==3.1.17->llmware) (4.8.0)\n",
      "Requirement already satisfied: tzdata in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from psycopg==3.1.17->llmware) (2023.3)\n",
      "Requirement already satisfied: requests in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from Wikipedia-API==0.6.0->llmware) (2.31.0)\n",
      "Requirement already satisfied: filelock in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from huggingface-hub>=0.19.4->llmware) (3.13.1)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from huggingface-hub>=0.19.4->llmware) (2023.10.0)\n",
      "Requirement already satisfied: tqdm>=4.42.1 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from huggingface-hub>=0.19.4->llmware) (4.66.1)\n",
      "Requirement already satisfied: pyyaml>=5.1 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from huggingface-hub>=0.19.4->llmware) (6.0.1)\n",
      "Requirement already satisfied: packaging>=20.9 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from huggingface-hub>=0.19.4->llmware) (23.1)\n",
      "Requirement already satisfied: audioread>=2.1.9 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from librosa>=0.10.0->llmware) (3.0.1)\n",
      "Requirement already satisfied: scipy>=1.2.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from librosa>=0.10.0->llmware) (1.11.3)\n",
      "Requirement already satisfied: scikit-learn>=0.20.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from librosa>=0.10.0->llmware) (1.3.1)\n",
      "Requirement already satisfied: joblib>=0.14 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from librosa>=0.10.0->llmware) (1.3.2)\n",
      "Requirement already satisfied: decorator>=4.3.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from librosa>=0.10.0->llmware) (5.1.1)\n",
      "Requirement already satisfied: numba>=0.51.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from librosa>=0.10.0->llmware) (0.59.1)\n",
      "Requirement already satisfied: soundfile>=0.12.1 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from librosa>=0.10.0->llmware) (0.12.1)\n",
      "Requirement already satisfied: pooch>=1.1 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from librosa>=0.10.0->llmware) (1.8.1)\n",
      "Requirement already satisfied: soxr>=0.3.2 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from librosa>=0.10.0->llmware) (0.3.7)\n",
      "Requirement already satisfied: lazy-loader>=0.1 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from librosa>=0.10.0->llmware) (0.4)\n",
      "Requirement already satisfied: msgpack>=1.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from librosa>=0.10.0->llmware) (1.0.8)\n",
      "Requirement already satisfied: anyio<5,>=3.5.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from openai>=1.0.0->llmware) (4.0.0)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from openai>=1.0.0->llmware) (1.9.0)\n",
      "Requirement already satisfied: httpx<1,>=0.23.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from openai>=1.0.0->llmware) (0.23.3)\n",
      "Requirement already satisfied: pydantic<3,>=1.9.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from openai>=1.0.0->llmware) (2.6.4)\n",
      "Requirement already satisfied: sniffio in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from openai>=1.0.0->llmware) (1.3.0)\n",
      "Requirement already satisfied: dnspython<3.0.0,>=1.16.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from pymongo>=4.7.0->llmware) (2.6.1)\n",
      "Requirement already satisfied: sympy in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from torch>=1.13.1->llmware) (1.12)\n",
      "Requirement already satisfied: networkx in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from torch>=1.13.1->llmware) (3.2.1)\n",
      "Requirement already satisfied: jinja2 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from torch>=1.13.1->llmware) (3.1.2)\n",
      "Requirement already satisfied: regex!=2019.12.17 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from transformers>=4.36.0->llmware) (2023.12.25)\n",
      "Requirement already satisfied: safetensors>=0.4.1 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from transformers>=4.36.0->llmware) (0.4.2)\n",
      "Requirement already satisfied: idna>=2.8 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from anyio<5,>=3.5.0->openai>=1.0.0->llmware) (2.10)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from botocore<1.28.0,>=1.27.53->boto3==1.24.53->llmware) (2.8.2)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.25.4 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from botocore<1.28.0,>=1.27.53->boto3==1.24.53->llmware) (1.26.18)\n",
      "Requirement already satisfied: certifi in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from httpx<1,>=0.23.0->openai>=1.0.0->llmware) (2023.7.22)\n",
      "Requirement already satisfied: httpcore<0.17.0,>=0.15.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from httpx<1,>=0.23.0->openai>=1.0.0->llmware) (0.16.3)\n",
      "Requirement already satisfied: rfc3986<2,>=1.3 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from rfc3986[idna2008]<2,>=1.3->httpx<1,>=0.23.0->openai>=1.0.0->llmware) (1.5.0)\n",
      "Requirement already satisfied: llvmlite<0.43,>=0.42.0dev0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from numba>=0.51.0->librosa>=0.10.0->llmware) (0.42.0)\n",
      "Requirement already satisfied: platformdirs>=2.5.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from pooch>=1.1->librosa>=0.10.0->llmware) (3.10.0)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from pydantic<3,>=1.9.0->openai>=1.0.0->llmware) (0.6.0)\n",
      "Requirement already satisfied: pydantic-core==2.16.3 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from pydantic<3,>=1.9.0->openai>=1.0.0->llmware) (2.16.3)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from requests->Wikipedia-API==0.6.0->llmware) (3.2.0)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from scikit-learn>=0.20.0->librosa>=0.10.0->llmware) (3.2.0)\n",
      "Requirement already satisfied: cffi>=1.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from soundfile>=0.12.1->librosa>=0.10.0->llmware) (1.15.1)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from jinja2->torch>=1.13.1->llmware) (2.1.3)\n",
      "Requirement already satisfied: mpmath>=0.19 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from sympy->torch>=1.13.1->llmware) (1.3.0)\n",
      "Requirement already satisfied: pycparser in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from cffi>=1.0->soundfile>=0.12.1->librosa>=0.10.0->llmware) (2.21)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from httpcore<0.17.0,>=0.15.0->httpx<1,>=0.23.0->openai>=1.0.0->llmware) (0.14.0)\n",
      "Requirement already satisfied: six>=1.5 in c:\\users\\prash\\appdata\\roaming\\python\\python311\\site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.28.0,>=1.27.53->boto3==1.24.53->llmware) (1.16.0)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install llmware"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from llmware.setup import Setup\n",
    "from llmware.library import Library\n",
    "from llmware.prompts import Prompt, HumanInTheLoop\n",
    "from llmware.retrieval import Query\n",
    "from llmware.configs import LLMWareConfig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MongoDB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example requires you to have an instance of MongoDB running to store the processed contracts. The easiest way to do this is to use Docker Compose with our provided file `llmware/docker-compose_mongo_milvus.yaml`. More information on how to do this is available in our [documentation](https://github.com/llmware-ai/llmware/blob/main/docs/getting_started/working_with_docker.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Worker function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this function, we will do the following:\n",
    "1. Loading in contracts using the `Setup` class\n",
    "2. Adding them to a library using the `Library` class\n",
    "3. Locate MSA files using a text query on the first page of each document using the `Query` class\n",
    "4. Perform a text query to find all text containing \"termination\"\n",
    "5. Pass these text chunks as a source to our _dragon-yi-6b-gguf_ model with the `Prompt` class\n",
    "6. Ask the model the question \"What is the notice for termination for convenience?\"\n",
    "7. Output the model's response\n",
    "8. Save the model's state as a CSV with the `HumanInTheLoop` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def msa_processing():\n",
    "\n",
    "    local_path = Setup().load_sample_files()\n",
    "    agreements_path = os.path.join(local_path, \"AgreementsLarge\")\n",
    "\n",
    "    #   create a library with all of the Agreements (~80 contracts)\n",
    "    msa_lib = Library().create_new_library(\"msa_lib503_635\")\n",
    "    msa_lib.add_files(agreements_path)\n",
    "\n",
    "    #   find the \"master service agreements\" (MSA)\n",
    "    q = Query(msa_lib)\n",
    "    query = \"master services agreement\"\n",
    "    results = q.text_search_by_page(query, page_num=1, results_only=False)\n",
    "\n",
    "    #   results_only = False will return a dictionary with 4 keys:  {\"query\", \"results\", \"doc_ID\", \"file_source\"}\n",
    "    msa_docs = results[\"file_source\"]\n",
    "\n",
    "    #   load prompt/llm locally\n",
    "    model_name = \"llmware/dragon-yi-6b-gguf\"\n",
    "    prompter = Prompt().load_model(model_name)\n",
    "\n",
    "    #   analyze each MSA - \"query\" & \"llm prompt\"\n",
    "    for i, docs in enumerate(msa_docs):\n",
    "\n",
    "        print(\"\\n\")\n",
    "        print (i+1, \"Reviewing MSA - \", docs)\n",
    "\n",
    "        #   look for the termination provisions in each document\n",
    "        doc_filter = {\"file_source\": [docs]}\n",
    "        termination_provisions = q.text_query_with_document_filter(\"termination\", doc_filter)\n",
    "\n",
    "        #   package the provisions as a source to a prompt\n",
    "        sources = prompter.add_source_query_results(termination_provisions)\n",
    "\n",
    "        print(\"update: sources - \", sources)\n",
    "\n",
    "        #   call the LLM and ask our question\n",
    "        response = prompter.prompt_with_source(\"What is the notice for termination for convenience?\")\n",
    "\n",
    "        #   post processing fact checking\n",
    "        stats = prompter.evidence_comparison_stats(response)\n",
    "        ev_source = prompter.evidence_check_sources(response)\n",
    "\n",
    "        for i, resp in enumerate(response):\n",
    "            print(\"update: llm response - \", resp)\n",
    "            print(\"update: compare with evidence- \", stats[i][\"comparison_stats\"])\n",
    "            print(\"update: sources - \", ev_source[i][\"source_review\"])\n",
    "\n",
    "        prompter.clear_source_materials()\n",
    "\n",
    "    # Save jsonl report with full transaction history to /prompt_history folder\n",
    "    print(\"\\nupdate: prompt state saved at: \", os.path.join(LLMWareConfig.get_prompt_path(),prompter.prompt_id))\n",
    "\n",
    "    prompter.save_state()\n",
    "\n",
    "    # Generate CSV report for easy Human review in Excel\n",
    "    csv_output = HumanInTheLoop(prompter).export_current_interaction_to_csv()\n",
    "\n",
    "    print(\"\\nupdate: csv output for human review - \", csv_output)\n",
    "\n",
    "    return 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Main block"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we call our worker function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "1 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Delta.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", ' of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products  of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\nof the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and  numbers indicated in this section. Notices will be deemed delivered \\nof the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and  numbers indicated in this section. Notices will be deemed delivered \\n'], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3620, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3620, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 401, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 401, 'evidence_stop_char': 803, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 803, 'evidence_stop_char': 1213, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1213, 'evidence_stop_char': 1623, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 2}\n",
      "update: llm response -  {'llm_response': '30 days written notice', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 1116, 'output': 5, 'total': 1121, 'metric': 'tokens', 'processing_time': 100.29986071586609}, 'time_stamp': '2024-06-21_162157', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3620, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3620, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf': [5]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "2 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Chi.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this  Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this  Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", ' of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)   year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products  of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)   year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\nof the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be deemed \\nof the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be deemed \\n'], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2011, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2011, 'evidence_stop_char': 2412, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2412, 'evidence_stop_char': 2814, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2814, 'evidence_stop_char': 3216, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3216, 'evidence_stop_char': 3618, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3618, 'evidence_stop_char': 4020, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 402, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 402, 'evidence_stop_char': 805, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 805, 'evidence_stop_char': 1206, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1206, 'evidence_stop_char': 1607, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 2}\n",
      "update: llm response -  {'llm_response': '30 days written notice', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this  Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this  Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 1116, 'output': 5, 'total': 1121, 'metric': 'tokens', 'processing_time': 106.03363370895386}, 'time_stamp': '2024-06-21_162343', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2011, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2011, 'evidence_stop_char': 2412, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2412, 'evidence_stop_char': 2814, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2814, 'evidence_stop_char': 3216, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3216, 'evidence_stop_char': 3618, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3618, 'evidence_stop_char': 4020, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf': [5]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Chi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "3 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Mu.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", 'any provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties  below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)   year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)   year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n'], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2815, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2815, 'evidence_stop_char': 3216, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3216, 'evidence_stop_char': 3617, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3617, 'evidence_stop_char': 4019, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 401, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 401, 'evidence_stop_char': 802, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 802, 'evidence_stop_char': 1205, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1205, 'evidence_stop_char': 1608, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 2}\n",
      "update: llm response -  {'llm_response': '30 days written notice.', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 1116, 'output': 6, 'total': 1122, 'metric': 'tokens', 'processing_time': 120.17748379707336}, 'time_stamp': '2024-06-21_162544', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2815, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2815, 'evidence_stop_char': 3216, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3216, 'evidence_stop_char': 3617, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3617, 'evidence_stop_char': 4019, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf': [5, 6]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice.', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice.', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Mu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "4 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Psi.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n\"], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Psi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 805, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Psi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 805, 'evidence_stop_char': 1207, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Psi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1207, 'evidence_stop_char': 1608, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Psi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1608, 'evidence_stop_char': 2009, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Psi.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2009, 'evidence_stop_char': 2412, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Psi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2412, 'evidence_stop_char': 2814, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Psi.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 1}\n",
      "update: llm response -  {'llm_response': '30 days written notice.', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 766, 'output': 6, 'total': 772, 'metric': 'tokens', 'processing_time': 86.09340858459473}, 'time_stamp': '2024-06-21_162710', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Psi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 805, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Psi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 805, 'evidence_stop_char': 1207, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Psi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1207, 'evidence_stop_char': 1608, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Psi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1608, 'evidence_stop_char': 2009, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Psi.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2009, 'evidence_stop_char': 2412, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Psi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2412, 'evidence_stop_char': 2814, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Psi.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Psi.pdf': [5, 6, 4]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice.', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Psi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice.', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Psi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "5 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Rho.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n\"], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Rho.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 805, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Rho.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 805, 'evidence_stop_char': 1207, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Rho.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1207, 'evidence_stop_char': 1608, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Rho.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1608, 'evidence_stop_char': 2011, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Rho.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2011, 'evidence_stop_char': 2412, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Rho.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2412, 'evidence_stop_char': 2814, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Rho.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 1}\n",
      "update: llm response -  {'llm_response': '30 days written notice.', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 766, 'output': 6, 'total': 772, 'metric': 'tokens', 'processing_time': 90.99819278717041}, 'time_stamp': '2024-06-21_162841', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Rho.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 805, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Rho.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 805, 'evidence_stop_char': 1207, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Rho.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1207, 'evidence_stop_char': 1608, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Rho.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1608, 'evidence_stop_char': 2011, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Rho.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2011, 'evidence_stop_char': 2412, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Rho.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2412, 'evidence_stop_char': 2814, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Rho.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Rho.pdf': [5, 6, 4]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice.', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Rho.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice.', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Rho.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "6 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Beta.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", ' of  the Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)   year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products  of  the Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)   year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\nof the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be deemed \\nof the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be deemed \\n'], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3620, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3620, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 401, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 401, 'evidence_stop_char': 803, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 803, 'evidence_stop_char': 1204, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1204, 'evidence_stop_char': 1605, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 2}\n",
      "update: llm response -  {'llm_response': '30 days written notice', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 1116, 'output': 5, 'total': 1121, 'metric': 'tokens', 'processing_time': 124.40906977653503}, 'time_stamp': '2024-06-21_163046', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3620, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3620, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf': [5]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "7 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Eta.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", ' of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products  of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be \\n   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be \\n'], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2815, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2815, 'evidence_stop_char': 3216, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3216, 'evidence_stop_char': 3619, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3619, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 401, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 401, 'evidence_stop_char': 803, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 803, 'evidence_stop_char': 1205, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1205, 'evidence_stop_char': 1607, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 2}\n",
      "update: llm response -  {'llm_response': '30 days written notice', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 1116, 'output': 5, 'total': 1121, 'metric': 'tokens', 'processing_time': 122.25001740455627}, 'time_stamp': '2024-06-21_163249', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2815, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2815, 'evidence_stop_char': 3216, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3216, 'evidence_stop_char': 3619, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3619, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf': [5]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Eta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "8 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Alpha.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", 'below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will  of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)   year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\nbelow.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)   year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n'], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3620, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3620, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 404, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 404, 'evidence_stop_char': 807, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 807, 'evidence_stop_char': 1212, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1212, 'evidence_stop_char': 1615, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 2}\n",
      "update: llm response -  {'llm_response': '30 days written notice', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 1116, 'output': 5, 'total': 1121, 'metric': 'tokens', 'processing_time': 106.45800447463989}, 'time_stamp': '2024-06-21_163435', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3620, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3620, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf': [5]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Alpha.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "9 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Phi.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices \\n\", ' below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices any provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)   year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)   year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n'], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2815, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2815, 'evidence_stop_char': 3216, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3216, 'evidence_stop_char': 3618, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3618, 'evidence_stop_char': 4019, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}], [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 400, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 400, 'evidence_stop_char': 802, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 802, 'evidence_stop_char': 1205, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1205, 'evidence_stop_char': 1608, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 2}\n",
      "update: llm response -  {'llm_response': '30 days written notice.', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 1116, 'output': 6, 'total': 1122, 'metric': 'tokens', 'processing_time': 108.62869024276733}, 'time_stamp': '2024-06-21_163624', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2815, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2815, 'evidence_stop_char': 3216, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3216, 'evidence_stop_char': 3618, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3618, 'evidence_stop_char': 4019, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf': [5, 6]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice.', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice.', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Phi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "10 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Nu.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices \\n below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices \\n\", 'any provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties any provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)   year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)   year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n'], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3619, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3619, 'evidence_stop_char': 4020, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}], [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 401, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 401, 'evidence_stop_char': 803, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 803, 'evidence_stop_char': 1206, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1206, 'evidence_stop_char': 1609, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 2}\n",
      "update: llm response -  {'llm_response': '30 days written notice.', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices \\n below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 1116, 'output': 6, 'total': 1122, 'metric': 'tokens', 'processing_time': 123.69110679626465}, 'time_stamp': '2024-06-21_163828', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3619, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3619, 'evidence_stop_char': 4020, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf': [5, 6]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice.', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice.', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Nu.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "11 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Gamma.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", ' of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products  of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be deemed \\n Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be deemed \\n'], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3620, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3620, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 401, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 401, 'evidence_stop_char': 803, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 803, 'evidence_stop_char': 1210, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1210, 'evidence_stop_char': 1617, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 2}\n",
      "update: llm response -  {'llm_response': '30 days written notice', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 1116, 'output': 5, 'total': 1121, 'metric': 'tokens', 'processing_time': 102.44451832771301}, 'time_stamp': '2024-06-21_164011', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3620, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3620, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf': [5]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Gamma.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "12 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Pi.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  yea r from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n\"], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Pi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 805, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Pi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 805, 'evidence_stop_char': 1207, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Pi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1207, 'evidence_stop_char': 1609, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Pi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1609, 'evidence_stop_char': 2011, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Pi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2011, 'evidence_stop_char': 2412, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Pi.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2412, 'evidence_stop_char': 2815, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Pi.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 1}\n",
      "update: llm response -  {'llm_response': '30 days written notice.', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  yea r from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 768, 'output': 6, 'total': 774, 'metric': 'tokens', 'processing_time': 67.6502456665039}, 'time_stamp': '2024-06-21_164119', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Pi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 805, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Pi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 805, 'evidence_stop_char': 1207, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Pi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1207, 'evidence_stop_char': 1609, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Pi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1609, 'evidence_stop_char': 2011, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Pi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2011, 'evidence_stop_char': 2412, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Pi.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2412, 'evidence_stop_char': 2815, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Pi.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Pi.pdf': [5, 6, 4]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice.', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Pi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice.', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Pi.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "13 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Zeta.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)   year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n\"], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Zeta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 805, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Zeta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 805, 'evidence_stop_char': 1207, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Zeta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1207, 'evidence_stop_char': 1609, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Zeta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1609, 'evidence_stop_char': 2010, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Zeta.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2010, 'evidence_stop_char': 2412, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Zeta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2412, 'evidence_stop_char': 2815, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Zeta.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 1}\n",
      "update: llm response -  {'llm_response': '30 days written notice.', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n below.   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)   year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 766, 'output': 6, 'total': 772, 'metric': 'tokens', 'processing_time': 64.77527284622192}, 'time_stamp': '2024-06-21_164224', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Zeta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 805, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Zeta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 805, 'evidence_stop_char': 1207, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Zeta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1207, 'evidence_stop_char': 1609, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Zeta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1609, 'evidence_stop_char': 2010, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Zeta.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2010, 'evidence_stop_char': 2412, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Zeta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2412, 'evidence_stop_char': 2815, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Zeta.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Zeta.pdf': [5, 6, 4]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice.', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Zeta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice.', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Zeta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "14 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Delta.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", ' of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products  of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\nof the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and  numbers indicated in this section. Notices will be deemed delivered \\nof the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and  numbers indicated in this section. Notices will be deemed delivered \\n'], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3620, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3620, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 401, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 401, 'evidence_stop_char': 803, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 803, 'evidence_stop_char': 1213, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1213, 'evidence_stop_char': 1623, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 2}\n",
      "update: llm response -  {'llm_response': '30 days written notice', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 1116, 'output': 5, 'total': 1121, 'metric': 'tokens', 'processing_time': 111.97353863716125}, 'time_stamp': '2024-06-21_164416', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3620, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3620, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf': [5]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Delta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "15 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Kappa.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", ' of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products    Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be \\n   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n'], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3620, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3620, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 401, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 401, 'evidence_stop_char': 803, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 803, 'evidence_stop_char': 1205, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1205, 'evidence_stop_char': 1607, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 2}\n",
      "update: llm response -  {'llm_response': '30 days written notice', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 1116, 'output': 5, 'total': 1121, 'metric': 'tokens', 'processing_time': 118.6111249923706}, 'time_stamp': '2024-06-21_164615', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3620, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3620, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf': [5]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Kappa.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "16 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Omega.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", '  Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  yea r from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  yea r from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n'], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3620, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3620, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 400, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 400, 'evidence_stop_char': 801, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 801, 'evidence_stop_char': 1204, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1204, 'evidence_stop_char': 1607, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 2}\n",
      "update: llm response -  {'llm_response': '30 days written notice', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 1116, 'output': 5, 'total': 1121, 'metric': 'tokens', 'processing_time': 116.53554511070251}, 'time_stamp': '2024-06-21_164811', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3620, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3620, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf': [5]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omega.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "17 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Epsilon.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", '   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be    Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n'], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2815, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2815, 'evidence_stop_char': 3216, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3216, 'evidence_stop_char': 3619, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3619, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 401, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 401, 'evidence_stop_char': 803, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 803, 'evidence_stop_char': 1205, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1205, 'evidence_stop_char': 1607, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 2}\n",
      "update: llm response -  {'llm_response': '30 days written notice', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 1116, 'output': 5, 'total': 1121, 'metric': 'tokens', 'processing_time': 117.87502908706665}, 'time_stamp': '2024-06-21_165009', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2815, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2815, 'evidence_stop_char': 3216, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3216, 'evidence_stop_char': 3619, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3619, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf': [5]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "18 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Epsilon.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", '   Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be    Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n'], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2815, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2815, 'evidence_stop_char': 3216, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3216, 'evidence_stop_char': 3619, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3619, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 401, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 401, 'evidence_stop_char': 803, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 803, 'evidence_stop_char': 1205, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1205, 'evidence_stop_char': 1607, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 2}\n",
      "update: llm response -  {'llm_response': '30 days written notice', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 1116, 'output': 5, 'total': 1121, 'metric': 'tokens', 'processing_time': 120.23511362075806}, 'time_stamp': '2024-06-21_165210', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2815, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2815, 'evidence_stop_char': 3216, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3216, 'evidence_stop_char': 3619, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3619, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf': [5]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Epsilon.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "19 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Omicron.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", ' of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products  of  the  Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)  year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\n  Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be \\n  Each of the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be \\n'], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2815, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2815, 'evidence_stop_char': 3216, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3216, 'evidence_stop_char': 3619, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3619, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 401, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 401, 'evidence_stop_char': 803, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 803, 'evidence_stop_char': 1204, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1204, 'evidence_stop_char': 1605, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 2}\n",
      "update: llm response -  {'llm_response': '30 days written notice', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,  the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be   illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 1116, 'output': 5, 'total': 1121, 'metric': 'tokens', 'processing_time': 111.38872861862183}, 'time_stamp': '2024-06-21_165401', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2815, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2815, 'evidence_stop_char': 3216, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3216, 'evidence_stop_char': 3619, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3619, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf': [5]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Omicron.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "\n",
      "20 Reviewing MSA -  C:\\Users\\prash\\llmware_data\\tmp\\parser_tmp\\process_pdf_files\\Testco MSA Beta.pdf\n",
      "update: sources -  {'text_batch': [\"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", ' of  the Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)   year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products  of  the Customer, which  includes  trade  secrets.   Accordingly,  in  order  to  prevent TestCo  and  its  employees  from  intentionally  or  unintentionally   misappropriating any residual Confidential Information, TestCo agr ees that for the period of one (1)   year from the termination of this Agreement, its employees who provide services hereunder will not   work on any Competitive Products \\nof the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be deemed \\nof the contracting parties agrees to notify the other in writing of address or  contact changes.     All  notices,  authorizations,  and  requests  given  or  made  in  connection  with  this  agreement,  including notice of   termination of this agreement, must be sent by  post, express courier, facsimile,  or email to the addresses and   numbers indicated in this section. Notices will be deemed \\n'], 'metadata_batch': [[{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3620, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3620, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 401, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 401, 'evidence_stop_char': 803, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 4, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 803, 'evidence_stop_char': 1204, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1204, 'evidence_stop_char': 1605, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 6, 'doc_id': 1, 'block_id': 1}]], 'batches_count': 2}\n",
      "update: llm response -  {'llm_response': '30 days written notice', 'prompt': 'What is the notice for termination for convenience?', 'evidence': \"must be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\nmust be   brought within two years from the date that the cause of action arose.   Term and Termination of Agreement; Assignment.   This Agreement shall remain in effect until terminated. Either party may terminate this agreement,  any Statement of   Work or Services Description for convenience by giving the other party 30 days   written  notice.  Either  party  may   terminate  this  Agreement  or \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n party  may  be   deemed  notice  of   termination  of  this  Agreement,  effective  on  the  date  of  assignment, by the other party.   Survival upon Termination or End of Term.   The   provisions  regarding  warranty,  limitation  of  liability,  confidentiality,  fees  and   expenses,  obligations  on   termination or expiration, ownership and license, and miscellaneous of this  Agreement, and \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n any  work  order  or  services  description  if  the  other  party  is  in  material  breach  or   default  of any obligation that is not cured  within 15 days' notice of such breach. The TestCo agrees to pay all fees   for services performed  and  expenses  incurred  prior  to  the  termination  of  this  Agreement.  Termination  of  this   Agreement  will  terminate  all  outstanding  Statement \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\n(24) months  following   termination  of  this  Agreement,  directly  or  indirectly,  call  on  or  attempt  to  call  on,  hire,  solicit,  or  induce  any   change  in  or  cessation  of,   the  business  relationship,  of  any   customers,  clients, contractors, vendors, contract   manufacturers, suppliers, investors or employees of other on  whom  the party  called  on  or  became  acquainted \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\nany provisions specified  as surviving in a Statement of Work or Services  Description, survive any termination or expiration of this agreement,   any Statement of Work or   Services Description.   Severability.  If  a  court  holds  any   provision  of  this  Agreement  to  be  illegal,  invalid  or  unenforceable,  the   remaining   provisions will remain in full force and effect and the parties \\n\", 'instruction': 'default_with_context', 'model': 'dragon-yi-6b-gguf', 'usage': {'input': 1116, 'output': 5, 'total': 1121, 'metric': 'tokens', 'processing_time': 112.91939091682434}, 'time_stamp': '2024-06-21_165554', 'calling_app_ID': '', 'rating': '', 'account_name': 'llmware', 'prompt_id': '1b84f914-d851-4a83-8b13-47cd464d92db', 'batch_id': 0, 'evidence_metadata': [{'batch_source_id': 0, 'evidence_start_char': 0, 'evidence_stop_char': 403, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 1, 'evidence_start_char': 403, 'evidence_stop_char': 806, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 2, 'evidence_start_char': 806, 'evidence_stop_char': 1208, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 3, 'evidence_start_char': 1208, 'evidence_stop_char': 1610, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 4, 'evidence_start_char': 1610, 'evidence_stop_char': 2012, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 5, 'evidence_start_char': 2012, 'evidence_stop_char': 2414, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 6, 'evidence_start_char': 2414, 'evidence_stop_char': 2816, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 7, 'evidence_start_char': 2816, 'evidence_stop_char': 3218, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 8, 'evidence_start_char': 3218, 'evidence_stop_char': 3620, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}, {'batch_source_id': 9, 'evidence_start_char': 3620, 'evidence_stop_char': 4022, 'source_name': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}], 'biblio': {'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf': [5]}, 'event_type': 'inference', 'human_feedback': '', 'human_assessed_accuracy': '', 'comparison_stats': {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}, 'source_review': [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]}\n",
      "update: compare with evidence-  {'percent_display': '100.0%', 'confirmed_words': ['30', 'days', 'written', 'notice'], 'unconfirmed_words': [], 'verified_token_match_ratio': 1.0, 'key_point_list': [{'key_point': '30 days written notice', 'entry': 0, 'verified_match': 1.0}]}\n",
      "update: sources -  [{'text': 'Services Description for convenience by giving the other party 30 days written notice Either party may terminate this Agreement or ', 'match_score': 1.0, 'source': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\tmp\\\\parser_tmp\\\\process_pdf_files\\\\Testco MSA Beta.pdf', 'page_num': 5, 'doc_id': 1, 'block_id': 1}]\n",
      "\n",
      "update: prompt state saved at:  C:\\Users\\prash\\llmware_data\\prompt_history\\1b84f914-d851-4a83-8b13-47cd464d92db\n",
      "\n",
      "update: csv output for human review -  {'report_name': 'interaction_report_2024-06-21_165554.csv', 'report_fp': 'C:\\\\Users\\\\prash\\\\llmware_data\\\\prompt_history\\\\interaction_report_2024-06-21_165554.csv', 'results': 20}\n"
     ]
    }
   ],
   "source": [
    "if __name__ == \"__main__\":\n",
    "\n",
    "    m = msa_processing()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our output contains a dictionary for each LLM response with several keys, notably:\n",
    "- `llm_response`: giving us the answer to our question\n",
    "- `evidence`: giving us the text where the model found the answer to the question"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
